The present invention relates generally to data storage and data backup systems, and more particularly, to systems and techniques which allow a file to be backed-up and restored between a data storage system and a backup server using concurrently transmitted trails of data from the same file.
The ever increasing popularity in the use of computer networks and computerized data processing systems for both recreational and commercial use has created vast amounts of data that must be stored. To serve this need, data storage system providers have developed mass data storage systems that use large high speed disk arrays to store and access data. In a typical data storage system using modern disk array technology, there are many individual physical storage devices, such as hard disk drives (disks), which provide storage, access, and management of the data. Remote computer systems such as hosts on a computer network that require access (read and/or write capability) to the data stored in the data storage system generally reference the data as one or more files within a filesystem.
Software applications that execute on the remote computer systems are generally responsible for generating the data that gets stored in the files within a data storage system. For example, a software application such as Microsoft Exchange produced by Microsoft Corporation of Redmond Wash. (Microsoft and Microsoft Exchange are trademarks of Microsoft Corporation) can be used to create and manage a large database of information that is stored as a single file within a data storage system. Depending upon the amount of data that is stored in the database, it is not uncommon for the size of the file containing the data to become quite large and exceed, for example, many tens of gigabytes in size.
A single hard disk within a data storage system cannot typically provide enough storage capacity by itself to store files and/or filesystems that grow to such large capacities. Accordingly, data storage systems often provide logical storage entities called xe2x80x9cvolumes.xe2x80x9d An administrator of the data storage system configures individual volumes during an initial data storage system setup procedure. A single volume of data, often referred to as a xe2x80x9clogical volume,xe2x80x9d provides a mechanism to associate many portions of different individual disks (or many entire disks ) together as a contiguous amount of available data storage space. A volume allows a large file that exceeds the size of any one hard disk drive to be stored across different portions of the disks that make up the volume in the data storage system. A process called a Logical Volume Manager (LVM) typically executes on a host and interfaces with process(s) and/or control circuitry within the data storage system to manage associated disks and data stored within volumes. The Logical Volume Manager also provides an interface for computer application programs to access (read and/or write) the files stored within the disks associated with the volume.
A portion of a hard disk drive that can be included within a volume is typically called a partition, and a hard disk may contain more than one partition. For example, during a disk configuration process, an administrator might xe2x80x9cpartitionxe2x80x9d a single ten gigabyte disk into two non-overlapping five gigabyte partitions. It is possible to configure the entire capacity of a single disk as a one partition (e.g., one ten gigabyte hard disk may be partitioned as one ten gigabyte partition). Each partition on a hard disk includes disk extent information which indicates how large the partition is (e.g. how many gigabytes) and where (e.g. between what disk addresses) the partition exists on the disk.
Volumes within a data storage system are thus a set of one or more disk partitions within a data storage system, presented by an LVM process to software applications on host(s) as a contiguous amount of storage space available to store files. As a specific example, a one hundred gigabyte volume may be comprised of twenty, five-gigabyte partitions, with each partition existing on a different disk.
Generally, two types of volumes exist in data storage systems: concatenated volumes and striped volumes. The specific type of a volume reflects how data is arranged for storage across the disk partitions that make up the volume. In concatenated volumes, data is stored sequentially. Thus, if a concatenated volume is composed of three disk partitions which are distributed across various disk drives on the data storage system, as a remote computer system stores (i.e. writes) more and more data to the volume, the LVM first will fill the first partition in the concatenated volume with data, followed by the second partition, and then the third partition. The LVM thus fills partitions in a concatenated volume from first to last in a sequential manner.
The LVM treats striped volumes somewhat differently than concatenated volumes. A striped volume also is a set of partitions distributed across one or more disks in a data storage system. Striped volumes have data written in incremental and equal amounts to each partition in the volume, a little at a time (whereas concatenated volumes fill one partition completely before beginning to fill the next in the volume). Accordingly, when storing data to a striped volume, the LVM writes small portions or segments of data evenly, to each disk partition associated with the striped volume. As such, each partition in the striped volume is filled equally, a little bit at a time. If the partition layout of a striped volume is diagramed, the data appears to be striped across each partition, hence the name striped volume. When using striped volumes, data is essentially spread evenly across each partition as the volume fills-up.
Referring now to another area related to the invention, it is quite common for computer systems to perform periodic backups of files stored in a data storage system. Backing-up files typically involves transferring a copy of the file stored in the disks within a data storage system to another medium, such as magnetic tape. Backing-up files protects against data loss in the event of disk or data storage system failures.
In conventional backup systems, backups of an entire volume are typically performed on a file by file basis. That is, each file within a volume is backed-up to a tape drive in sequential order. Thus, if a volume contains three files A, B and C, file A might be backed up to tape first, followed by file B and then by file C. During a conventional backup operation, data storage systems provide the data from each file for backup purposes from beginning to end, just as if each file were being sequentially loaded in its entirety into memory in a host computer system. For example, to backup file A, the LVM starts by reading data from the beginning of file A, and then continually reads and streams the data to the backup storage system (e.g., a tape drive unit) until the end of the A file is reached. Files B and then C would be performed thereafter in successive order. If the LVM uses concatenated volumes, the LVM begins reading file A from the first partition of the volume containing this file, and if file A spans more than one partition, once the LVM completes streaming data for file A from the first partition storing file A, the LVM proceeds to the next (i.e. second) partition that stores the next portion of file A, and so on until the file is completely backed up from the concatenated volume.
To backup a file stored on a striped volume, the LVM sequentially reads small segments of data from each partition. The first segment of data read from each of the first to the last partition in the volume forms the first stripe of the striped volume. When the first stripe has been read, the LVM returns to the first partition to begin reading of the second stripe, and the process repeats by reading a small segment of data from each partition until the second strip is completely read. The process continues until the LVM has read all of the data for the entire file from the striped partition. Note that in striped volumes, just as in concatenated volumes, the LVM sequentially reads and streams the data from a single file stored on a striped volume to the backup device from beginning to end. When the backup of file A is completed, the backup process then repeats for files B and C, at which point the backup is complete and files A, B and C are stored to magnetic tape.
More than one tape might be required to store the contents of files A, B and C, depending upon each file""s size and the total size of all of the data to be backed-up. However, each file is read from the data storage system as a single stream or xe2x80x9ctrailxe2x80x9d of data from the beginning of the file to its end, and each file is stored on tape as a single contiguous backup image of the file. For example, a single tape might be able to store up to forty-two gigabytes of data. If file A is fifty gigabytes, and files B and C are only 10 gigabytes each, two tapes are required to backup files A, B and C. During execution of a conventional backup process, the backup device (e.g. tape drive) sequentially reads file A, placing the first forty-two gigabytes onto the first tape. Then, after the second tape is mounted, the remaining eight gigabytes of file A are stored to the beginning of the second tape. Thereafter, the backup device sequentially reads file B, storing it on the next ten gigabytes of tape 2, followed by file C which is backed up to the next ten gigabytes of tape 2. In this manner, the backup device stores each file by receiving a single data stream.
To restore a file that has been previously backed up to tape, a conventional restore operation is generally performed in reverse order of the prior art backup operation. During a restore, a backup system mounts the tape containing the beginning of the file to be restored and reads and streams data from the backup image of the file on tape to the LVM which places the data onto storage in the data storage system. The stream of data is received as a single data trail from the beginning to the end of the file being restored. If the file on tape was originally stored in a concatenated volume, then the LVM simply begins writing the stream of data to a partition within the volume to which the file is being restored in the data storage system. Once the first partition fills up, the LVM advances to the next partition of the concatenated volume and continues to write the stream of data. This process continues until the file is completely restored. If the file is restored to a striped volume, then the LVM sequentially writes small predetermined segments of the data stream to each partition as they arrive in a serial fashion. When the last partition of the volume is read for the first time, the first stripe of data for the file is complete and the LVM returns to write the next segment received to begin the second stripe on the volume. The LVM thus controls the striping of the file to various partitions during the restore operation, just as if the data were being received from an application creating (i.e., writing to disk) the file in its entirety for the first time.
Prior art backup processes that use conventional techniques to backup and restore files to and from a data storage system suffer from a number of deficiencies. Most notably, conventional backup solutions backup entire individual files, which may be quite large in size, onto tape as a single complete backup file image. That is, each file in a prior art backup system is backed up as a singular work item and resides on the backup medium (e.g. tape) as a sequentially contiguous file from start to end. There are no provisions in prior art backup (or restore) systems to backup a single file as a group of smaller entities or portions. Accordingly, backup operations can take inordinate amounts of time to complete, especially for very large files.
For example, a typical data storage system can provide a stream of data from a single hard disk drive at a speed of approximately ten megabits per second. Interestingly, if two streams of data are read from the same disk, the speed of each trail drops dramatically down to about two and a half megabits per second, due to the overhead involved in seeking data on the disk for both streams. Since prior art backup systems backup a file from the beginning to the end of the file as a single stream or trail of data, the fastest data rate that can be provided for that singular stream is ten megabits per second (i.e., the data rate of data being read from the hard disk using one stream). In other words, since the file is backed up sequentially from start to end, only one stream of data is available to provide data from the file to the backup medium.
Similarly, a commonly used backup mechanism such as a magnetic tape drive is relatively slow, and can only accept and write data at a rate of approximately five to seven megabits per second. When backing up instances of large files, the speed of the backup might be further limited by the speed of the tape drive accepting the single stream of data for a file. More advanced backup servers might provide many tape drives that can be used simultaneously. However, since individual files are backed up with a single stream of data, even with multiple files being backed up to different tape drives at the same time, the backup operation is restricted to taking at least as long as the time required to backup the largest file using one stream of data for that file. Thus, multiple tape drives that can be run in unison might be helpful to speed the backup process of different files (e.g., with each tape drive backing up a different file), but the problem of lengthy backup time still remains when one file is very large and is limited to being backed up with the single stream of data.
Similar problems exist in prior art restore systems. When doing a prior art restore of a file, since the file was backed up as one large sequential stream or trail of data, the restore system must read the file from the backup tape device in a similar manner. Only one stream of data is thus available at any point in time for any individual file being restored from a backup device to a data storage device. Just to restore a large file, the restore operation may take many minutes or hours to complete. This is unacceptable in highly reliable computer data processing systems that require lost data to be restored quickly.
The present invention is based in part on the observation of the aforementioned problems with prior art backup and restore systems and does much to overcome and minimize such problems. The present invention provides a system including mechanisms and techniques to allow a file to be backed up from a data storage system to a backup device using multiple trails or streams of data. In general, the system of the invention allows a data storage system to concurrently or simultaneously provide many streams of data for the same file during a backup operation. Each data stream corresponds to data read from a different portion of the file. The invention thereby allows the entire operation of backing up a file to be done in portions that are concurrently backed up in parallel with each other. This significantly reduces the time required to backup the file.
In a similar manner, the invention also provide mechanisms and techniques to restore, from a backup device to a data storage system, a single file using multiple trails or streams of data. Since portions of the file are received over different streams of data that are transferred to the data storage system in parallel with each other, restoring a large file in this manner takes considerably less time than conventional prior art file restoring techniques.
More specifically, according to one aspect of the invention, a system of the invention provides a technique for backing up a file. The system implements the technique by first determining which respective data portions of the file are stored upon which respective storage devices of a plurality of storage devices. In an example system configuration, the data portions of the file are partitions of the storage devices, which are hard disk drives within a data storage system. The system then designates individual respective backup devices of a plurality of backup devices to be used for backing-up respective individual ones of the data portions of the file that are stored on respective storage devices. The backup devices in a typical configuration of the invention are tape drives using tape media for storing the backup data. The process of designating thus matches or pairs-up different data portions of the file to respective different tapes on the tape drives (i.e., backup devices). The technique of the invention then concurrently backs-up at least two respective data portions of the file (e.g., two different partitions) to different respective backup devices for which those data portions are designated, such that the at least two respective data portions of the file are backed-up to respective backup devices in parallel. For instance, with two tape drives, two data portions of the same file could be backed up at the same time. In this manner, the invention is able to simultaneously backup different data portions of the same file in a concurrent fashion, thus saving significant amounts of time when backing up the file.
In somewhat more detail, to determining which respective data portions of the file are stored upon which respective storage devices, the system of the invention performs the operations of obtaining an identity of a volume that stores the respective data portions of the file. This may done, for instance, by querying the Logical Volume Manager. Next, the system obtains, from the volume, disk extent information for each respective data portion of the file, the disk extent information identifying, for each respective data portion, a respective storage device that stores the data portion and a size and placement of the data portion on the storage device. Once the size and placement of each data portion of the file are known, the system creates a list of backup work items, each backup work item in the list corresponding to one of the respective data portions of the file to be backed-up and including the disk extent information for that respective data portion. Essentially, as will be discussed in more detail, a list of backup work items indicates what data portions from a hard disk are to be written to what tape drives in the backup storage system.
The file to be backed-up may be comprised of potentially many different data portions on different disk partitions. To create a list of backup work items, the system of the invention in one arrangement assigns a unique respective backup file name for each respective data portion of the file. The system then associates the disk extent information (e.g., size, placement, etc.) for each respective data portion of the file to the unique respective backup file name assigned to that respective data portion. Accordingly, in this particular arrangement of the invention, the list of backup work items contains a new file name (which is essentially treated as a file to be backed up, in and of itself) for each data portion of the file, and for each new file name, the system assigns the extent information for a respectively associated data portion as well as a respective backup device (e.g. tape drive) to the file name.
In one arrangement of the system of the invention, to designate which individual respective backup devices (e.g., tapes) of a plurality of backup devices are to be used for backing-up respective individual ones of the data portions (e.g., partitions), the system sequentially selects and assigns backup work items from the list of backup work items to a backup device sequentially selected from the plurality of backup devices. In a preferred arrangement, backup work items that correspond to data portions that reside on a common storage device (e.g., two partitions of the same file on a single hard disk drive) are assigned to the same backup device (e.g. tape). In a more specific implementation of this technique, the system assigns respective data portions of the file that are stored on different storage devices to different respective backup devices, such that during the operation of concurrently backing-up the at least two respective data portions of the file, no two data portions that reside on a common storage device are read at the same time.
This technique ensures that if two data partitions which store the single file to be backed up are on the same disk, then those partitions will be streamed to the same tape drive but at different times (e.g., one after another). Or in other words, the technique of concurrently backing-up at least two respective data portions of the file concurrently backs up a plurality of data portions of the file in parallel to a plurality of respectively assigned backup devices, such that no two data portions that exist on a common storage device are backed up in parallel.
One arrangement of the invention sequentially selects and assigns backup work items to a backup device by determining if a data portion corresponding to a currently selected backup work item is stored on the same storage device as a backup work item that has been previously selected and that has been assigned to a backup device. If this is the case, then the system assigns the currently selected backup work item to the same storage device as the backup work item previously selected. By repeating the steps of sequentially selecting, assigning and determining in a round-robin manner until all backup work items have been assigned to a specific respective backup device, the system of the invention can ensure that data portions from the same disk are paired-up or assigned for backup (e.g., in the backup work item list) to the same tape. Once the backup work item list is created, the technique of the invention concurrently backs up a plurality of data portions of the file (one per backup work item) in parallel to a plurality of respectively assigned backup devices, while ensuring that no two data portions that exist on a common storage device are backed up in parallel at the same time.
According to another arrangement of the system of the invention, if a tuning file (to be explained later) is available which specifies which data portions are to be backed-up to which backup devices, the system uses assignments between backup work items and backup devices as specified in the tuning file. If no tuning file is present however, the system performs the previous operations of sequentially selecting, and assigning, and determining until all backup work items have been assigned to a respective backup device.
To perform the operation of concurrently backing-up each data portion, the technique of the invention concurrently performs, for each respective backup device having at least one designated data portion to backup, a backup operation of one data portion designated to be backed-up by that backup device. Once the invention detects completion of the backup operation of each respective backup device, the system determines if other designated data portions remain to be backed up, and if so, repeats the techniques explained above of concurrently performing a backup operation and detecting completion of the backup operation for any designated data portions that remain to be backed up.
During the concurrent backup operation of the system of the invention, the system receives a first stream of data from a first data portion of the file stored within a first data storage device within the data storage system and writes the first stream of data to a first respective backup device for which the first data portion is designated. Concurrently therewith, the system also receives a second stream of data from a second data portion of the file stored within a second data storage device within the data storage system and writes the second stream of data to a second respective backup device for which the second data portion is designated. The operations of receiving the first stream of data and receiving the second stream of data are performed concurrently such that different respective data portions of the file are simultaneously received and written to different backup devices in the data backup system. As indicated above, this vastly improves file backup times.
These techniques may be applied to any size file stored on any number of disks in any number of data portions, though benefits are more pronounced for large files. According to one arrangement of the invention, prior to backing up the data portions, the system detects if the file to be backed-up exceeds a predetermined threshold which indicates that a multiple trail file backup is to be performed. If such a determination is made, the system then performs the steps of determining, designating and concurrently backing-up as outlined above. The predetermined threshold may be, for example, an amount of time required to completely backup the file using a non-multiple trail file backup method or may be a maximum total size of the backup file. The file to be backed up may be included in a list of files to be backed up by the backup server and the system may perform the procedure outlined above for each file within the list of files.
Another technique encompassed within the system of the invention provides a technique for preparing a list of backup work items for a backup server. The list contains a list of backup work items (one per data portion of a file) to be backed-up. To create such a list, the technique retrieves a list of files to be backed up, and, for each single file in the list of files, determines if the single file exceeds a predetermined size threshold. If the size threshold is met, the system employing the technique determines if the single file meets criteria for a multiple trail file backup. The criteria may be, for example, that the file is stored as at least two distinct data portions on two distinct storage devices such as hard disks, and that there are at least two distinct backup devices such as tape drives to perform a backup operation once the list of backup work items is complete. If the system determines that the size threshold and multiple trail criteria are met, the technique of the system obtains an identity of a volume that stores respective data portions of the file and then obtains, from the volume, disk extent information for each respective data portion of the file. The disk extent information identifies, for each respective data portion, a respective storage device that stores the data portion and a size and placement of the data portion on the storage device. The technique of the system of the invention then creates a list of backup work items, with each backup work item in the list corresponding to one of the respective data portions of the file to be backed-up and each including the disk extent information for that respective data portion. In this manner, each data portion of the file can be treated, according to this technique of the system of the invention, as a separate work item during a subsequent backup process.
The invention also includes a method of providing data from a data storage system to a backup server. In an arrangement supporting this method, the system provides, as output from the data storage system, a first stream of data from a first portion of a file stored on a first data storage device within the data storage system. The system also provides, as output from the data storage system, a second stream of data from a second portion of the file stored on a second data storage device within the data storage system. The operations of providing a first stream of data and providing a second stream of data are performed concurrently between a backup client program and a backup server program such that different respective data portions of the file are simultaneously provided as output from the data storage system to the backup server. This aspect of the invention allows multiple trails or streams of data to be read from the data storage system that are all part of the same file to be backed up, thus increasing the efficiency of the data storage system in providing data from a file to a backup server.
The system of the invention also provides a method for restoring a file using multiple trails provided from backup devices, such as tape drives, from within the backup server. According to this technique, the system determines which respective data portions (i.e., different backup images on tape of different partitions from disk(s) that originally stored the file) of the file are stored upon which respective backup devices (i.e., tape drives) of a plurality of backup devices (e.g., a set of tape drives in a backup server). The system then designates individual respective portions (i.e., partitions) of storage devices (i.e., hard disk drives or other storage media) of a plurality of storage devices to be used for storing respective individual ones of the data portions of the file that are stored on respective backup devices. The system then concurrently restores at least two respective data portions (i.e., backup file images on different tapes) of the file to different respective storage devices (i.e., to partitions on respective disks) for which those data portions are designated, such that at least two respective data portions of the file are restored to respective storage devices from respective backup devices in parallel.
The system of the invention itself includes a backup server computer system comprising a backup client interface, a backup device interface coupled to a plurality of backup devices, a processor, and an interconnection mechanism coupled to the backup client interface, the storage device interface and the processor. A memory is provided and is coupled to the interconnection mechanism. The memory stores a backup server program. The backup server program is executable on the processor to cause the processor to perform the methods and techniques outlined above.
Specifically, in one arrangement, the backup server program determines which respective data portions of a file to be backed-up are stored upon which respective storage devices of a plurality of storage devices accessible by a host backup client. The backup server program designates individual respective backup devices of the plurality of backup devices to be used for backing-up respective individual ones of the data portions of the file that are stored on respective storage devices. Next, the backup server program concurrently backs-up at least two respective data portions of the file to different respective backup devices for which those data portions are designated, such that respective data portions of the file are backed-up to respective backup devices in parallel via the backup server computer system.
The backup server system can also restore files using multiple trails in order to cut down the time required to perform file restore operations. According to this arrangement of the backup server, the backup server program is executable on the processor to cause the processor to determine which respective data portions of a file are stored upon which respective backup devices of the plurality of backup devices. Then, the program designates individual respective portions of storage devices of the plurality of storage devices accessible via the host backup client to be used for storing respective individual ones of the data portions of the file that are stored on respective backup devices. Next, the program causes the processor to concurrently restore at least two respective data portions of the file to different respective storage devices for which those data portions are designated. The concurrent restore is performed such that the at least two respective data portions of the file are restored to respective storage devices accessible to the host backup client from respective backup devices interfaced to the backup server in parallel.
Another arrangement of the backup server comprises a plurality of storage devices and a backup client interface to couple the backup server to at least one host backup client. A processor is provided and is coupled to the plurality of storage devices and to the backup client interface. The processor executes a series of instructions to allow the backup server to backup a single file that is stored as a plurality of data portions across a plurality of storage devices accessible to the host backup client. In this arrangement, each data portion is backed up as a stream of data to a respective backup device and at least two streams of data are received for backup in parallel to the backup server from the host backup client.
Other arrangements of the invention that are disclosed herein include software programs to perform the backup and restore operations summarized above. More particularly, a computer program product is disclosed which has a computer-readable medium including computer program logic encoded thereon as a backup server program for backing up a file. The computer program logic, when executed on at least one processing unit with a backup server, causes the processing unit to perform the operations of backing up and restoring as indicated herein. This arrangement of the invention is typically provided as software on a computer readable medium such as an optical, floppy or hard disk or other such medium such as firmware in a ROM or RAM chip. The software can be installed onto a computer to perform the techniques explained herein. For instance, a backup client computer can install the software which then executes to periodically backup files to the backup server, which also executes one or more software programs that perform the backup server methods of the invention.